Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Novel Methods of Identifying Individual and Neighborhood Risk Factors for Loss to Follow-Up After Ophthalmic Screening.

PRCIS: Residence in a middle-class neighborhood correlated with lower follow-up compared with residence in more affluent neighborhoods. The most common explanations for not following up were the process of making an appointment and lack of symptoms. PURPOSE: To explore which individual-level and neighborhood-level factors influence follow-up as recommended after positive ophthalmic and primary care screening in a vulnerable population using novel methodologies. PARTICIPANTS AND METHODS: From 2017 to 2018, 957 participants were screened for ophthalmic disease and cardiovascular risk factors as part of the Real-Time Mobile Teleophthalmology study. Individuals who screened positive for either ophthalmic or cardiovascular risk factors were contacted to determine whether or not they followed up with a health care provider. Data from the Social Vulnerability Index, a novel virtual auditing system, and personal demographics were collected for each participant. A multivariate logistic regression was performed to determine which factors significantly differed between participants who followed up and those who did not. RESULTS: As a whole, the study population was more socioeconomically vulnerable than the national average (mean summary Social Vulnerability Index score=0.81). Participants whose neighborhoods fell in the middle of the national per capita income distribution had a lower likelihood of follow-up compared with those who resided in the most affluent neighborhoods (relative risk ratio=0.21, P -value<0.01). Participants cited the complicated process of making an eye care appointment and lack of symptoms as the most common reasons for not following up as instructed within 4 months. CONCLUSIONS: Residence in a middle-class neighborhood, difficulty accessing eye care appointments, and low health literacy may influence follow-up among vulnerable populations.

Environmental influences on ophthalmic conditions: A scoping review.

BACKGROUND: Environmental factors have been implicated in various eye pathologies. The purpose of this review is to synthesise the published research on environmental effects on eye disease. METHODS: Four databases were searched for terms relating to environmental exposures and ophthalmic disease. Titles and abstracts were screened followed by full-text review. Data was extracted from 118 included studies. Quality assessment was conducted for each study. RESULTS: Air pollutants, including nitrogen dioxide, nitrites, sulphur dioxide, particulate matter, carbon monoxide, ozone and hydrocarbons are associated with ocular conditions ranging from corneal damage to various retinopathies, including central retinal artery occlusion. Certain chemicals and metals, such as cadmium, are associated with increased risk of age-related macular degeneration. Climate factors, such as sun exposure, have been associated with the development of cataracts. Living in rural areas was associated with various age-related eye diseases whereas people living in urban settings had higher risk for dry eye disease and uveitis. CONCLUSION: Environmental exposures in every domain are associated with various ophthalmic conditions. These findings underscore the importance of continued research on the interplay between the environment and eye health.

A Review of Cost-Effectiveness Analyses for Open Angle Glaucoma Management.

PURPOSE: Cost-effectiveness analyses (CEAs) quantify and compare both costs and measures of efficacy for different interventions. As the costs of glaucoma management to patients, payers, and physicians are increasing, we seek to investigate the role of CEAs in the field of glaucoma and how such studies impact clinical management. METHODS: We adhered to the "Preferred Reporting Items for Systematic Reviews and Meta-analyses" guidelines for our systematic review structure. Eligible studies included any full-text articles that investigated cost-effectiveness or cost-utility as it relates to the field of open angle glaucoma management in the United States. Risk of bias assessment was conducted using the validated Joanna Briggs Institute Critical Appraisal Checklist for Economic Evaluations. RESULTS: Eighteen studies were included in the review. Dates of publication ranged from 1983 to 2021. Most of the studies were published in the 2000s and performed CEAs in the domains of treatment/therapy, screening, and adherence for patients with primary angle open glaucoma. Of the 18 articles included, 14 focused on treatment, 2 on screening, and 2 on adherence. Most of these studies focused on the cost-effectiveness of different topical medical therapies, whereas only a few studies explored laser procedures, surgical interventions, or minimally invasive procedures. Economic models using decision analysis incorporating state-transition Markov cycles or Montecarlo simulations were widely used, however, the methodology among studies was variable, with a wide spectrum of inputs, measures of outcomes, and time horizons used. CONCLUSION: Overall, we found that cost-effectiveness research in glaucoma in the United States remains relatively unstructured, resulting in unclear and conflicting implications for clinical management.

Cost-effectiveness of Prophylactic Laser Peripheral Iridotomy in Primary Angle-Closure Suspects.

PURPOSE: To investigate the cost-effectiveness (CE) of prophylactic laser peripheral iridotomy (LPI) in primary angle-closure (PAC) suspects (PACSs). DESIGN: Cost-effectiveness analysis utilizing Markov models. SUBJECTS: Patients with narrow angles (PACSs). METHODS: Progression from PACSs through 4 states (PAC, PAC glaucoma, blindness, and death) was simulated using Markov cycles. The cohort entered at 50 years and received either LPI or no treatment. Transition probabilities were calculated from published models, and risk reduction of LPI was calculated from the Zhongshan Angle Closure Prevention trial. We estimated costs of Medicare rates, and previously published utility values were used to calculate quality-adjusted life year (QALY). Incremental CE ratios (ICER) were evaluated at $50 000. Probabilistic sensitivity analyses (PSAs) addressed uncertainty. MAIN OUTCOME MEASURES: Total cost, QALY, and ICER. RESULTS: Over 2 years, the ICER for the LPI cohort was > $50 000. At 6 years, the LPI cohort was less expensive with more accrued QALY. In PSA, the LPI arm was cost-effective in 24.65% of iterations over 2 years and 92.69% over 6 years. The most sensitive parameters were probability of progressing to PAC and cost and number of annual office visits. CONCLUSIONS: By 6 years, prophylactic LPI was cost-effective. The rate of progressing to PAC and differing practice patterns most impacted CE. With uncertainty of management of narrow angles, cost may be a decision management tool for providers. FINANCIAL DISCLOSURE(S): The authors have no proprietary or commercial interest in any materials discussed in this article.

Validation of the New York University Langone Eye Test Application, a Smartphone-Based Visual Acuity Test.

Purpose: To validate and assess user satisfaction and usability of the New York University (NYU) Langone Eye Test application, a smartphone-based visual acuity (VA) test. Design: Mixed-methods cross-sectional cohort study. Participants: Two hundred forty-four eyes of 125 participants were included. All participants were adults 18 years of age or older. Participants' eyes with a VA of 20/400 (1.3 logarithm of the minimum angle of resolution [logMAR]) or worse were excluded. Methods: Patients were tested using the clinical standard Rosenbaum near card and the NYU Langone Eye Test application on an iPhone and Android device. Each test was performed twice to measure reliability. Ten patients were selected randomly for subsequent semistructured qualitative interviews with thematic analysis. Main Outcome Measures: Visual acuity was the parameter measured. Bland-Altman analysis was used to measure agreement between the results of the NYU Langone Eye Test application and Rosenbaum card, as well as test-retest reliability of each VA. The correlation between results was calculated using the intraclass correlation coefficient. Satisfaction survey and semistructured interview questions were developed to measure usability and acceptability. Results: Bland-Altman analysis revealed an agreement between the application and the Rosenbaum near card of 0.017 ± 0.28 logMAR (iPhone) and 0.009 ± 0.29 logMAR (Android). The correlation between the application and the Rosenbaum near card was 0.74 for both the iPhone and Android. Test-retest reliability was 0.003 ± 0.22 logMAR (iPhone), 0.01 ± 0.25 logMAR (Android), and 0.01 ± 0.23 logMAR (Rosenbaum card). Of the 125 participants, 97.6% found the application easy to use, and 94.3% were overall satisfied with the application. Thematic analysis yielded 6 key themes: (1) weaknesses of application, (2) benefits of the application, (3) tips for application improvement, (4) difficulties faced while using the application, (5) ideal patient for application, and (6) comparing application with traditional VA testing. Conclusions: The NYU Langone Eye Test application is a user-friendly, accurate, and reliable measure of near VA. The application's integration with the electronic health record, accessibility, and easy interpretation of results, among other features, make it ideal for telemedicine use.

Cost-Effectiveness Analysis of Minimally Invasive Trabecular Meshwork Stents with Phacoemulsification.

PURPOSE: To investigate the costs and effects of implanting trabecular meshwork bypass stents during cataract surgery from a societal perspective in the United States. DESIGN: Cost-utility analysis using Markov models and efficacy/safety data from published pivotal or randomized control trials (RCTs) of devices investigated. PARTICIPANTS: Patients aged 65 years and older with mild to moderate primary open-angle glaucoma with or without visually significant cataract. METHODS: With the use of Markov models, glaucoma progression through 4 glaucoma states (mild, moderate, advanced, severe/blind) and death were simulated over 35 years. The cohort with cataract entered the model and received cataract surgery with or without device implantation. We included a medication management only reference group to calculate total costs and outcomes for those without cataract. Intraocular pressure (IOP) reductions from RCTs were converted to glaucoma state transition probabilities using visual field (VF) mean deviation (MD) decline rates from the Early Manifest Glaucoma Trial. Progressive thinning of the retinal nerve fiber layer (RNFL) on OCT imaging related to IOP control warranted further intervention, including adding medication, selective laser trabeculoplasty (SLT), or incisional glaucoma surgery. We estimated whole costs at Medicare rates and obtained utility values for glaucoma states from previous studies. Incremental costs per quality-adjusted life-year (QALY) gained were evaluated at a QALY threshold of $50 000. One-way deterministic sensitivity analysis, scenario analyses, and probabilistic sensitivity analyses addressed parameter uncertainty and demonstrated model robustness. MAIN OUTCOME MEASURES: Total costs, QALY, and incremental cost-effectiveness ratio (ICER). RESULTS: Over 35 years in the base case, the Hydrus (Ivantis, Inc.) implanted with cataract surgery arm cost $48 026.13 and gained 12.26 QALYs. The iStent inject (Glaukos Corp.) implanted with cataract surgery arm cost $49 599.86 and gained 12.21 QALYs. Cataract surgery alone cost $54 409.25 and gained 12.04 QALYs. Initial nonsurgical management cost $57 931.22 and gained 11.74 QALY. The device arms dominated or were cost-effective compared with cataract surgery alone within 5 years and throughout sensitivity analyses. The iStent inject arm was cost-effective in 94.19% of iterations in probabilistic sensitivity analyses, whereas the Hydrus arm was cost-effective in 94.69% of iterations. CONCLUSIONS: Implanting the Hydrus Microstent or iStent inject during cataract surgery is cost-effective at a conservative QALY threshold.